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We report on a first search for production of the lightest neutral CP-even Higgs boson (h) in the 
next-to-minimal supersymmetric standard model, where h decays to a pair of neutral pseudoscalar 
Higgs bosons (a), using 4.2 fb _1 of data recorded with the DO detector at Fermilab. The a bosons 
are required to either both decay to fi + \i~ or one to fi + pT and the other to t + t~ . No significant 
signal is observed, and we set limits on its production as functions of M a and Mh- 

PACS numbers: 12.60.Fr, 14.80.Cp 



The CERN e+e" Collider (LEP) has excluded a SM- 
like Higgs boson decaying to bb, t + t~ with a mass below 
114.4 GeV [l[, resulting in fine-tuning being needed in 
the minimal supersymmetric SM (MSSM). Slightly richer 
models, such as the next-to-MSSM (NMSSM) 0, allevi- 



ate this fine-tuning 0. The h^bb branching ratio (BR) 
is greatly reduced because the h dominantly decays to 
a pair of lighter neutral pseudoscalar Higgs bosons (a). 
The most general LEP search yields M h > 82 GeV Q, 
independent of the Higgs boson decay. 
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Helicity suppression causes the a boson to decay to 
the heaviest pair of particles kinematically allowed. The 
BR(o-^MM) is nearly 100% for 2m /J <M a <3m 7r («450 
MeV) and then decreases with rising M a due to decay 
into hadronic states [B[. A M(fifi) spectrum in E de- 
cays consistent with a— where M a — 214.3 MeV was 
reported by the HyperCP collaboration J6J , which sug- 
gests searching for h— >aa with a— >fj,[i, [7|. Decays to 
charm are usually suppressed in the NMSSM, so they 
have been neglected. If 2m T <M a <2mi >) the BR(a^/i/j) 
is suppressed by {M*/M?)/[y/i - (2M T /M a ) 2 }, a decays 
primarily to t + t~, and the limit from LEP is still weak 
{M h >86 GeV) The direct search for the 4r final state 
is challenging, due to the lack of an observable resonance 
peak and low e, /j, transverse momentum (pr) which com- 
plicates triggering The 2/x2r final state however, con- 
tains a resonance from a— >fi[i, high px muons for trigger- 
ing, and missing transverse energy ($ T ) [13]. B-factories 
also search for T— ►a'y, where the a boson escapes as miss- 
ing energy or decays to muons or taus fill ]. 

In this Letter, we present a first search for h boson 
production, followed by h—*aa decay with either both 
a bosons decaying to or one decaying to /i+ 'fi~ 

and the other to t + t~. Data from Run II of the Fermi- 
lab Tevatron Collider recorded with the DO detector [l2| 
are used, corresponding to an integrated luminosity of 
about 4.2 fb _1 . The signal signature is either two pairs 
of collinear muons (due to the low M a ), or one pair of 
collinear muons and either large f^p, an additional (not 
necessarily isolated) muon, or a loosely-isolated electron 
from a—>TT opposite to the muon pair. The main back- 
grounds are multijet events containing muons from the 
decay of particles in flight (ir, K), heavy-flavor decays, 
and other sources (rj, </>, J/tp, etc.) and Z /7*(— >/zju)+jets. 
The pythia [TH event generator is used to simulate 
gg~>h^aa signal events for various Mh and M ai which 
are then passed through the GEANT3 14[ DO detector 
simulation and reconstructed. 

Events are required to have at least two muons recon- 
structed in the muon system and matched to tracks from 
the inner tracking system with pt > 10 GeV and M<2, 
where r\ is the pseudorapidity. Muons are not required 
to have opposite electric charge. No specific trigger re- 
quirements are made; an OR of all implemented triggers 
is used. But most events selected pass a dimuon trigger, 
with muon px thresholds of 4-6 GeV. Trigger efficiency 
is >90% for events passing the offline selections. 

For the 4/x channel, we look for one muon from each 
of the two a boson decays, so the dimuon pair with the 
largest invariant mass is selected, with M(/i,/i)>15 GeV 
and AK(n,fi)>l, where ATZ = ^(A-g) 2 + (Ac/)) 2 and (j> is 
the azimuthal angle. Only one muon is required to be re- 
constructed from each pair of collinear muons. The muon 
system has insufficient granularity to reliably reconstruct 
two close muons. A companion track is identified with 
Pt>4 GeV and smallest ATZ from each muon, within 
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FIG. 1: The y m2(/i, track) vs. y mi (p, track) distribution 
(a) in the multijet sample, and (b) after the isolation cut is 
applied to both muons for data and various MC signal masses. 



A1Z<1 and Az(track,PV)<l cm, where z is the distance 
along the beamline, and PV is the primary pp interaction 
vertex. The muon pair calorimeter isolation (2^) is the 
sum of calorimeter energy within 0.1<A7<!.<0.4 of either 
the muon or the companion track. Both muons are re- 
quired to have ZfL<l GeV and track-based isolation: <3 
tracks with px>0-5 GeV and Az(track,PV)<l cm within 
ATZ<0.5 of the muon, including the muon track itself. 

Based on a control data sample greatly enhanced in 
multijet events by removing the Z?„ requirement on the 
muons, we predict 1.9±0.4 events to pass the final selec- 
tions. The mass of the leading (trailing) px muon and its 
companion track, mi (ji, track) (m2(/i, track)), is shown 
in the multijet sample in Fig. [lja) and is used to model 
the background shape. Background is also expected from 
Z /^j*— events where additional companion tracks are 
reconstructed. Studying the dimuon mass distributions 
in the isolated data when zero or one of the muons is 
required to have a companion track gives an estimate of 
0.29±0.04 events. The background from it, diboson, and 
VF+jets production is found to be negligible. 

Signal acceptance uncertainty is dominated by the abil- 
ity to simulate the detection of the companion track, par- 
ticularly when the two muons are very collinear. Wc 
compare K§ decays in data and simulation as a function 
of the ATZ between the two pion tracks. Over most of 
the ATZ range, the relative tracking efficiency is within 
20%, but few events have A7?.<0.02 (corresponding to 
M a <0.5 GeV for Mft=100 GeV), and consistency can 
only be confirmed at the 50% level. For A7?.(/i,/i)<0.1 
(corresponding to M a <2 GeV for M h =100 GeV), there 
is the possibility that the two muons will overlap in the 
muon system and interfere with each other's proper re- 
construction and triggering. By studying the effect of 
adding noise hits, we find up to a 10% effect on recon- 
struction and 20% effect on the trigger efficiency. The 
background uncertainty (50%) is dominated by the sta- 
tistical uncertainty of the multijet-enhanced data sample. 
The luminosity uncertainty is 6.1% [HI]. 

After the isolation requirements are applied to both 
muons, two events are observed in data, consistent with 
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TABLE I: The efficiency for MC signal events within the 
2 s.d. window around each M a , numbers of events ex- 
pected from background (with statistical uncertainty) and 
observed in data, and the expected and observed limits on 
the (r(pp->h+X)xBR(h->aa-t4ij), for M h =100 GeV. Limits 
for other M a , up to 2m T , are interpolated from these simu- 
lated MC samples. No events are observed in a window for 
any interpolated M a . 



M a 


Window 


Eff. 


Abckg 


Nobs 


ctxBR 


(GeV) 


(MeV) 








[exp] obs (fb) 


0.2143 


±15 


17% 


0.001±0.001 





[10.0] 10.0 


0.3 


±50 


16% 


0.006±0.002 





[9.5] 9.5 


0.5 


±70 


12% 


0.012±0.004 





[7.3] 7.3 


1 


±100 


13% 


0.022±0.005 





[6.1] 6.1 


3 


±230 


14% 


0.005±0.002 





[5.6] 5.6 



the total background of 2.2±0.5 events. Neither has a 
third muon identified, compared to about 50% of the sig- 
nal MC events. We fit a Gaussian distribution to the 
mi (/i, track) distribution, and the number of events with 
both mi (/it, track) and rri2(/z, track) within a ±2 s.d. win- 
dow around the mean from the fit are determined for 
data, signal, and background (Tab. [J). No events are ob- 
served within any window, in agreement with the back- 
ground prediction. Upper limits on the h-^aa^Afi signal 
rate are computed at 95% C.L. using a Bayesian tech- 
nique [lj| and vary slightly with Mh, decreasing by «10% 
when Mh increases from 80 to 150 GeV. 

For the 2/x2t channel, the muon pair is selected in each 
event with the largest scalar sum of muon pt (S^ T ), with 
muon PT >10 GeV, A7£(/x, fi)<l, and Af(/z^)<20 GeV. 
This is the "pre-selection" (Tab. HI). Next, ££ T >35 GeV 
is required, to reduce background, and the same muon 
pair calorimeter and track isolation cuts are applied as 
for the channel. This is the "isolated" selection. 

Standard DO r identification [I3| is severely degraded 
and complicated by the topology of the two overlapping 
r leptons. Instead, we require significant f T from the 
collinear r decays to neutrinos. The $rp is computed from 
calorimeter cell energies and corrected for the pt of the 
muons. To ensure that this correction is as accurate as 
possible, the following additional muon selection criteria 
are applied. The muons' tracks in the inner tracker are 
required to have fits to their hits with x 2 /dof<4, trans- 
verse impact parameter from the PV less than 0.01 cm, 
and at least three hits in the silicon detector. The match 
between the track reconstructed from muon system hits 
and the track in the inner tracker must have x 2 <40, and 
the muon system track must have pr>8 GeV. Hits are 
required for both muons in all three layers of the muon 
system. Also, less than 10 GeV of calorimeter energy 
is allowed within A7?.<0.1 of either muon, to exclude 
muons with showers in the calorimeter. Finally, the lead- 
ing muon px must be less than 80 GeV, to remove muons 



with mismeasured pt- To improve the $ T measurement 
in the calorimeter, the number of jets reconstructed (l8j 
with cone radius 0.5, pr>15 GeV (corrected for jet en- 
ergy scale), and 1 77 1 < 2 . 5 must be less than five. Events 
with $r>80 GeV are also rejected to remove rare events 
where the $ T is grossly mismeasured, since signal is not 
expected to have such large These are the "refining" 
cuts. Then an event must pass one of three mutually 
exclusive subselections. The first subselection, for when 
no jet is reconstructed from the tau pair, requires zero 
jets with pt>15 GeV, A0(/i/i,$ T )>2.5, the highest-pr 
track with Az(track, PV)<3 cm and not matching either 
of the two selected muon tracks in the dimuon candidate 
to have p T >4 GeV and A</> (track, $ T )<0.7. The second 
subselection, for when at least one of the tau decays is 1- 
prong, requires at least one jet, where the lcading-p^ jet 
(jetl) has no more than four (non-muon) tracks associ- 
ated with it withpr>0.5 GeV, Az(track,jetl)<3 cm, and 
Aft(track,jetl)<0.5, A</>(jetl,$ T )<0.7, and $ T >20 GeV. 
The third subselection, for when both tau decays are 
3-prong (or more) and thus most jet-like, requires at 
least one jet, where jetl has either more than four (non- 
muon) tracks associated with it or A0(jetl,4r)>O-7 and 
$r>35 GeV. Events passing one of these three subselec- 
tions are called the u $r" selection. 

To gain acceptance, we also select events not passing 
the "4t" selection, but with either an additional muon 
(not necessarily isolated) or loosely-isolated electron. For 
the "Muon" selection, a (third) muon is required, with 
p T >4 GeV and A0(/x,$ T )<O.7. The "EM" selection re- 
jects events in the "Muon" selection and then requires 
an electron with pt>4 GeV, Acf)(e,$ T )<0.7, fewer than 
three jets, ( T >10 GeV, and pf,±^ r >35 GeV. 

The dimuon invariant mass shape of the multijet and 
7* background to the "^p" selection is estimated from 
the low frp data which passes the "refining" cuts but fails 
the "^r" selection cuts. For the "Muon" and "EM" se- 
lections, it is taken from the "isolated" data sample. The 
requirements of the "Muon" and "EM" selections have 
no significant effect on the dimuon invariant mass shape 
for a data sample with loosened isolation requirements. 
These background shapes are summed and normalized to 
the data passing all selections, but excluding data events 
within a 2 s.d. dimuon mass window for each M a (see 
below). Background from diboson, ti, and W±jets pro- 
duction, containing true $ T from neutrinos, is estimated 
using MC and found to contribute <10% of the back- 
ground from multijet and 7*. 

Signal acceptance uncertainty for the 2/x2r channel is 
dominated by the ability of the simulation to model the 
efficiency of the "refining" muon cuts and final selections. 
It is found to be 20% per-event based on studies of the 
muon and event quantities used, comparing data and MC 
events in the Z boson mass region. Comparing the J/tp 
and Z boson yields gives a 10% trigger efficiency uncer- 
tainty. The background uncertainty is less than 20% and 
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TABLE II: Selection efficiencies and limits for the 2\x2t channel, for Mh = 100 GeV and various M a . The numbers of events 
at "pre-selected," "isolated" stages and after ("refining") "Jfi T ," "Muon," and "EM" selections, assuming a(pp^h+X) — 1.9 pb 
and BR(ft— >aa)=l. Next are the window size, and numbers of events in the window for signal (and overall efficiency times 
BR), expected from background (with statistical uncertainty), and observed in data. The expected and observed limits on 
a(pp^h+X)x'BR(h^aa) and a(pp^h+X) xBR(/hoh) x 2 x BR(o->^)xBR(a^rr) follow. 



Sample 


N pre. 


N iso. 


(ref.) "$ T " 


"Mu" 


"EM" 


Window 


iVsig (Eff.) 


-/Vbckg 


iVobs 


[exp] obs 


a x 2xBR 


Data 


95793 


2795 


(1085) 15 


4 


4 














M a =3.6 GeV 


53.1 


28.0 


(14.5) 3.5 


1.9 


0.8 


±0.30 GeV 


5.2 (0.066%) 


1.9±0.4 


1 


[1.8] 1.5 pb 


[23.8] 19.1 fb 


M a =4 GeV 


33.6 


15.3 


(8.1) 2.5 


1.2 


0.4 


±0.32 GeV 


3.3 (0.042%) 


1.1±0.2 


4 


[2.6] 4.9 pb 


[23.9] 45.9 fb 


M a =7 GeV 


20.6 


8.7 


(4.5) 1.7 


0.8 


0.3 


±0.54 GeV 


2.1 (0.027%) 


1.1±0.2 


1 


[4.0] 3.9 pb 


[25.0] 24.6 fb 


M a =W GeV 


19.3 


7.5 


(4.2) 1.1 


0.6 


0.3 


±0.95 GeV 


1.5 (0.020%) 


1.6±0.3 


2 


[5.9] 6.5 pb 


[24.7] 27.3 fb 


M a =19 GeV 


14.6 


5.4 


(2.9) 0.8 


0.4 


0.2 


±1.37 GeV 


1.2 (0.015%) 


0.6±0.1 


1 


[6.3] 7.1 pb 


[30.0] 33.7 fb 




Data D0, 4.2 fb" 1 

Background 
Signals 
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FIG. 2: The dimuon invariant mass for events passing all 
selections in data, background, and 2/i2r signals for M a = 
3.6, 4, 7, 10, and 19 GeV. a(pp-^h+X) = 1.9 pb is assumed, 
BR{h^aa)=l, and Af h = 100 GeV. 
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FIG. 3: The expected and observed limits and ±1 s.d. and 
±2 s.d. expected limit bands for a(pp^h+X) xBR(/naa), 
for (a) M h =100 GeV and (b) M a =4 GeV. The signal 
for BR(/i— >aa)=l is shown by the solid line. The region 
M h <86 GeV is excluded by LEP. 



dominated by the statistical uncertainty of the data sam- 
ple used. Alternate fits of the background shape from low 
$ T data modify the background estimates by up to 10%. 

Figure [2] shows the dimuon invariant mass for data, 
background, and signals, after all selections. Each signal 
dimuon mass peak is fit to a Gaussian distribution, and 
the numbers of events with dimuon mass within a ±2 
s.d. window around the mean from the fit are counted 
(Tab. [TTJ) . Data in each window are consistent with the 



predicted background. The expected and observed limits 
on the crxBR of the h— >aa process for each M a studied 
are shown, assuming the a boson BRs given by PYTHIA, 
with no charm decays. Since the a boson BRs are model- 
dependent, we also derive a result which factors out the 
BRs taken from pythia. Limits are derived for interme- 
diate M a by interpolating the signal efficiencies and win- 
dow sizes, see Fig.[3][a). Above 9.5 GeV, we expect a—tbb 
decays to dominate and greatly decrease BR(aa— >2/j2t), 
but limits are calculated under the assumption that the b 
quark decays are absent. We also study the limits vs. Mh 
for M a = 4 GeV, see Fig. GP). 

We have presented results of the first search for 
Higgs boson production in the NMSSM decaying into 
a bosons at a high energy hadron collider, in the 
4fi and 2{i2t channels. The predicted BR(a^^/j) is 
driven at low M a by competition between decays to 
and to gluons and has large theoretical uncertain- 
ties [3]. Therefore, for M a <2m T , we set limits only 
on cr(p7j^/i±A")xBR(/i^aa)xBR 2 (a^^ + /i _ ), exclud- 
ing about 10 fb. Assuming a(pp^h+X)=1.9 pb [20j ]. 
corresponding to Mh~100 GeV, BR(a->fi/i) must there- 
fore be less than 7% to avoid detection, assuming a 
large BR(h-^aa). However, BR(a->/*/() is expected to 
be larger than 10% for M a <2m c 5], and depending on 
BR(a— >cc), which is model-dependent and typically sup- 
pressed in the NMSSM, could remain above 10% until 
M a =2m T . Thus these results severely constrain the re- 
gion 2m jl <M a <2m r . For M a >2m T , the limits set by 
the current analysis are a factor of «l-4 larger than the 
expected production cross section. 
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